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1.  INTRODUCTION 


A  national  goal  is  the  development  of  a  telesurgical  system  that  will  assist  a  medical  care 
provider  (who  might  have  limited  medical  care  experience)  in  the  management  of  a  medical 
emergency  (including  surgical  intervention)  in  a  location  which  is  remote  from  the  hospital 
and  the  experts  available  there.  Such  an  event  might  occur  in  both  civilian  and  non-civilian 
situations  in  locations  such  as  battlefields,  natural  disasters,  research  stations  in  the 
Antarctic,  and  NASA  space  missions  among  others.  Our  research  focuses  on  two 
interrelated  problems: 

1) .  Cooperative  manipulation  between  a  paramedic  local  to  the  patient  and 

a  surgeon  operating  through  a  telerobot. 

2) .  Coping  with  time  delay  in  both  audio/visual  and  force  feedback. 

1.1  BACKGROUND  AND  SIGNIFICANCE 

There  are  many  situations  where  an  acutely  and/or  critically  ill  patient  needs  to  be 
stabilized  and  cared  for  during  the  triage  and  transport  to  a  medical  center.  In  the  optimal 
case,  the  initial  care  of  the  patient  would  be  performed  by  a  physician  who  would  be  riding 
in  an  ambulance  enroute  to  a  hospital.  However,  necessary  care  in  remote  locations  might 
have  to  be  provided  by  individuals  who  have  minimal  or  no  patient  management  skills. 
Normally  under  such  circumstances  no  major  diagnostic  or  therapeutic  interventions  are 
advisable  during  the  initial  triage  at  the  accident  site,  nor  during  the  subsequent  transport  to 
the  major  medical  center.  We  are  proposing  a  telemedical  acute  care  consultant  system  that 
addresses  this  problem  in  an  effort  to  improve  patient  morbidity  and  mortality  in  these 
clinically  suboptimal  situations. 

1.2  TELESURGERY 

A  telediagnostic  communications  link  consists  of  a  (ideally  two  way)  closed-circuit 
video  system,  where  the  medical  care  consultant  can  pan/tilt/zoom  a  remote  camera  to 
observe  the  patient  on  a  video  monitor  (and  ideally  also  observe  the  local  medical  care 
provider  who  is  with  the  patient)  and  converse  over  a  synchronized  audio  link.  An 
augmented  system  allows  the  physician  to  remotely  position  surgical  instruments,  receive 
tactile  and  kinesthetic  feedback,  and,  with  the  manipulative  assistance  of  the  local  provider, 
perform  diagnostic  and  preliminary  surgical  intervention  on  the  patient. 

The  type  of  telesurgical  link  used  is  dependent  on  the  conununication  links  available  at 
the  site.  Alternatives  include  ISDN  (0.128  megabits/sec),  fractional  T1  (0.384 


megabits/sec),  T1  (1.544  megabits/sec),  T3  (45  megabits/sec),  satellite  transmission,  and 
microwave  link. 

There  are  various  barriers  to  implementation  of  telemedical  systems.  Foremost  is  the 
issue  of  liability  for  the  surgeon,  the  paramedic,  and  the  manufacturer  of  the  equipment. 
There  is  the  high  communication  cost  associated  with  long  distance  telecommunication  rates 
or  satellite  time,  in  addition  to  the  initial  costs  of  improving  outdated  equipment  and  phone 
lines.  And  in  most  cases,  there  are  no  standards  of  care  or  reimbursement  mechanisms  in 
place  for  the  telesurgical  services. 

1.3  COOPERATIVE  MANIPULATION 

We  define  cooperative  manipulation  to  be  the  physical  manipulation  of  the  patient 
mediated  by  audio-visual  communication  and  control  through  interaction  between 
individuals  working  toward  a  common  objective.  Three  parameters  are  important  in 
cooperative  manipulation:  conununication,  manipulation,  and  objective. 

In  our  project  the  parameters  are: 

Communication:  Because  our  interaction  will  be  between  a  paramedic  local  to  the 
site  and  a  surgeon  at  a  remote  medical  center,  communication  will  be  via  an  audio-video 
link  which  will  have  some  amount  of  time  delay.  If  the  distances  are  large  and  especially  if 
satellite  communication  is  used  that  delay  has  a  significant  effect  on  control  stability. 

Manipulation:  The  diagnostic  and  therapeutic  tools  that  will  be  used  are  the  DPL 
cannula,  laparoscopic  surgiports,  laparoscope,  and  laparoscopic  tools  (forceps,  shears, 
blunt  probe,  electrocautery,  and  clips). 

Objective:  The  initial  diagnosis  and  surgical  treatment  of  trauma  patients  during 
triage  and  transport  to  the  medical  center. 

To  our  knowledge,  no  one  has  previously  studied  the  interaction  between  a  human  and 
telemanipulator  controlled  by  another  human  remote  to  the  site. 


page  2 


2.  FIRST  YEAR  PROGRESS  REPORT 


2.1  HUMAN-IN-THE-LOOP  EXPERIMENTAL  TELESURGERY 

SIMULATION 

To  support  the  experiments  in  laparoscopic  telesurgery,  we  make  use  of  a  teleoperator 
system,  a  laparoscopic  surgical  trainer,  various  surgical  tools,  and  supporting  computer 
hardware.  Fig.  2.1  shows  the  system  configuration  including  the  human  participants  and 
the  major  apparatus  components.  The  apparatus  is  described  in  the  following  sections. 

2.1.1  Teleoperator  system: 

The  master-slave  system  that  we  use  is  composed  of  a  pair  of  PHANToM  haptic 
feedback  arms.  These  arms,  originally  designed  for  use  individually  in  providing  haptic 
feedback  for  virtual  environments,  are  ideal  in  a  bi-lateral  force-reflecting  teleoperator 
system  because  of  their  high  bandwidth  in  both  position  control  and  force  feedback, 
because  of  their  size  and  because  the  kinematics  of  each  arm  are  identical.  The  arms  (Figs. 
2.2a,  2.2b)  are  mounted  such  that  a  user  would  stand  with  his  or  her  shoulder  under  the 
first  axis,  giving  full  access  to  the  range  of  arm  motion. 

At  present,  both  arms  are  controlled  by  interface  cards  installed  in  a  single  Pentium  PC. 
It  is  expected  that  in  later  experiments,  the  cards  will  be  installed  in  separate  Pentiums 
linked  by  ethemet  or  some  other  high  bandwidth  conununications  medium.  In  this  way, 
the  arms  will  be  able  to  be  separated  by  arbitrary  distances.  To  evaluate  the  effect  of  time 
delays  on  control,  control  signals  will  be  buffered  for  set  periods,  then  transmitted  to  the 
slave. 

As  supplied  by  SensAble  Devices,  the  arms  provide  three  degrees  of  freedom  (DOF) 
with  force  feedback  for  endpoint  translation,  and  a  gimbal  wrist  with  orientation 
measurement,  but  no  torque  feedback  in  the  wrist  DOF.  For  use  in  laparoscopy,  the  roll 
axis  of  a  tool  must  also  be  controlled,  and  a  further  DOF  is  required  to  actuate  tools  like 
forceps  or  scissors.  The  novel  motorized  tool  designed  and  built  to  satisfy  these 
requirements  and  the  tool  handle  to  control  it  are  described  in  the  next  sections. 

2.1.2  Telesurgery  tool: 

The  basic  laparoscopy  tools  (e.g.  dissecting  and  grasping  forceps,  scissors)  have  two 
degrees  of  freedom  each;  one  to  control  the  roll  axis  of  the  shaft,  and  one  to  control  the 
open  and  close  motion  of  the  tool.  The  final  design  (Fig.  2.3)  uses  a  3W  DC  motor  and  a 
lead  screw  mechanism  to  drive  the  gripper  motion  and  a  .75  W  DC  motor  to  drive  a  pinion- 
gear  pair  for  the  roll  axis.  Position  measurement  is  achieved  through  optical  encoders 
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Fig.  2.1:  Telesurgery  experiment  configuration 
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Fig.  2  .2a:  Master  Manipulator 


Fig.  2.2b:  Slave  Manipulator 
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N 


Fig.  2.3:  Actuated  laparoscopy  tool  with  quick  change  tip 


mounted  to  the  motors  themselves.  Rather  than  designing  entirely  new  tools,  the  shafts 
and  tool  tips  from  laparoscopy  tools  manufactured  by  AutoSuture  were  modified  so  they 
could  be  driven  by  this  system.  The  axial  force  required  for  the  scissor  tool  (the  highest  for 
all  the  tools  tested)  is  on  the  order  of  SON,  higher  than  the  limit  for  the  motor,  so  thrust 
bearings  just  forward  of  the  3W  motor  gearhead  are  used  to  carry  the  load. 

The  tool  shafts  themselves  are  interchangeable  so  the  drive  portion  of  the  tool  can  be 
used  with  different  tool  tips.  Total  mass  is  approximately  220g  (approx.  8oz).  While  this 
is  higher  than  a  normal  unmodified  tool  (50-100g/2-4oz),  it  is  not  unreasonably  so,  and  the 
extra  weight  can  be  compensated  for  in  the  controller. 

Preliminary  tests  show  that  the  tool  performs  as  designed,  being  able  to  cut  through 
suture  material  or  several  thickness  of  paper  and  grasp  and  hold  objects  firmly. 
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2.1.3  Telesurgery  tool  handle: 

The  tool  handle  is  a  modified  EndoGrasp  handle.  A  short  length  of  the  original  shaft  is 
mounted  to  an  encoder  to  provide  roll  axis  measurements,  and  another  encoder  is  mounted 
on  one  side  to  measure  the  scissor  motion  (Fig.  2.4). 

The  EndoGrasp  handle  was  chosen  because  it  also  has  a  trigger  which  can  be 
instrumented  in  the  future  if  additional  inputs  are  required. 

At  present,  the  tool  handle  does  not  provide  force  feedback  for  either  the  roll  motion  or 
the  gripper.  However,  since  laparoscopic  surgeons  currently  depend  almost  entirely  on 
visual  feedback,  this  is  not  seen  as  a  major  disadvantage.  A  more  serious  disadvantage 
was  recognized  during  design:  without  torque  feedback  to  the  master,  the  handle  can  be 
oriented  in  directions  other  than  those  corresponding  with  the  slave  tool  axis.  This  might 
cause  confusion  for  the  surgeon.  To  alleviate  this  problem  without  the  necessity  of 
providing  full  6  DOF  force  feedback,  a  rod  will  be  attached  to  the  master  where  a  normal 
tool  shaft  would  be,  which  will  pass  through  a  cannula  mounted  in  the  same  position 
relative  to  the  master  as  the  real  working  cannula  is  to  the  slave  (Fig.  2.5).  In  this  way 


surgeon  will  be  subject  to  the  same  constraints  as  the  tool.  Given  that  most  feedback  in 
laparoscopy  is  visual,  minor  discrepancies  in  the  directions  of  force  feedback  due  to  slight 
misalignments  of  the  cannula  should  not  be  distracting  to  the  surgeon. 


2.1.4  Laparoscopy  trainer: 

The  trainer  (Fig.  2.6)  provides  a  stage  on  which  to  perform  our  experiments.  It 
includes  typical  features  found  in  basic  commercial  trainers:  an  opaque  cover  to  block 
direct  vision,  a  number  of  ports  through  which  cannulae  and  surgical  instruments  can  be 
inserted  and  a  stage  to  attach  test  apparatus.  The  walls  of  the  trainer  are  transparent  Lexan, 
so  an  external  video  camera  can  record  tests  from  a  fixed  viewpoint.  The  top  is  a  frame  of 
acrylic  strips  holding  pieces  of  1/8"  sheet  neoprene.  It  was  found  that  the  neoprene  has  a 
tendency  to  tear  if  a  cannula  inserted  directly  through  it  is  moved  even  small  amounts,  so 
the  ports  cut  into  the  rubber  are  covered  with  four  layers  of  (0.5  mm)  latex  rubber,  which 
does  not  tear  when  the  cannulae  are  moved.  The  trainer  is  approximately  15xl5"xl0" 
high. 

A  Polhemus  6-d.o.f.  position  sensor  can  be  used  inside  the  trainer.  The  magnetic  field 
emitter  unit  is  mounted  underneath  the  stage,  and  the  sensor  can  be  attached  to  the 
laparoscope,  various  tools,  or  a  dedicated  probe  (modified  clip  applier)  (Fig.  2.7).  This 
device  will  allow  tracking  of  tool  positions  within  the  trainer  in  case  these  data  are  needed. 
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Fig.  2.7:  Experimental  Stage.  Shows  Polhemus  probe, 
apparatus  for  laparoscopy  training  tasks 
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2.1.5  CCD  laparoscope  and  video  system: 

Surgeons  receive  visual  feedback  in  laparoscopy  through  the  use  of  a  laparoscope.  It  is 
a  tube,  10-1 1mm  in  diameter,  containing  fiber  optic  bundles  to  transmit  light  to  the 
operative  site  and  the  image  back  to  an  externally  mounted  video  camera.  In  our 
experiments  it  will  not  be  necessary  to  insert  or  remove  the  laparoscope,  so  a  simpler 
device  without  fiber  optics  was  constructed  to  serve  in  its  place. 

A  desktop  video  camera  was  modified  and  the  CCD  was  mounted  to  a  5/16"  acrylic 
rod.  An  extension  cable  was  constructed  to  connect  the  CCD  to  its  supporting  circuitry. 
The  rod  will  be  able  to  be  attached  to  the  motorized  tool  to  control  its  rotation  (Fig.  2.8). 

While  it  is  a  fixed  focus  device,  the  camera  has  a  small  aperature  so  the  depth  of  field  in 
focus,  which  has  been  set  to  approximately  2-8"  from  the  lens,  is  sufficient  for  these 
experiments.  The  camera  produces  a  color  image  with  a  resolution  of  330  horizontal  lines 
by  350  vertical.  The  main  circuit  board  for  the  camera  is  mounted  inside  the  laparoscopy 
trainer,  and  is  connected  to  the  CCD  by  a  16"  ribbon  cable. 
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Currently,  video  output  is  sent  through  an  SGI  Indigo^  workstation  to  a  20"  video 
monitor  seen  by  both  participants.  In  the  delay  experiments,  the  surgeon  will  have  a 
separate  monitor  which  receives  a  delayed  signal  through  an  audio/video  delay  device. 
This  artificial  delay  emulates  the  expected  communication  delay  in  a  real  system. 
Recording  can  be  done  from  two  points  of  view  with  the  use  of  our  lab  VCR  and 
camcorder. 

2.1.6  Computer  support: 

In  addition  to  the  Pentium  PC  running  the  PHANToM  arms,  we  are  using  an  SGI 
Indigo^  workstation  to  provide  support  for  the  video/graphics  presented  to  the  surgeon  and 
assistant  and  another  PC  (486)  to  sample  input  from  the  Polhemus  sensor. 

The  SGI  has  a  Galileo  video/graphics  card  installed  which  provides  the  ability  to 
combine  graphic  images  with  the  video  input  from  the  laparoscope  or  other  video  camera. 
This  gives  the  surgeon  access  to  one  extra  "tool"  in  addition  to  the  teleoperated  one— the 
mouse  pointer,  which  will  allow  the  surgeon  to  indicate  positions  in  the  field  of  view  in  an 
analog  fashion,  rather  than  describing  the  positions  verbally  to  the  assistant. 

The  three  machines  are  currently  linked  together  through  an  ethemet  LAN.  There  is  an 
internet  node  and  an  ISDN  phone  line  in  are  lab,  giving  us  two  immediate  options  for  the 
communications  mode  to  be  used  in  the  remote  experiments  and  demonstrations. 


2.2.  COPING  WITH  TIME  DELAY  FOR  THE  TELESURGERY  SYSTEM 

2.2.1.  Instability  of  Force  Reflecting  Control  in  Teleoperation  System: 

Several  successful  demonstrations  of  remotely  controlled  endoscopic  surgery  and 
laparoscopic  surgery  have  taken  place  recently  (Rininsland,1993;  Partin,  1995).  However, 
in  these  demonstrations  there  was  no  appreciable  time  delay  between  the  surgeon’s  actions 
and  the  resulting  change  in  the  endoscopic/  laparoscopic  image  or  between  the  surgeon’s 
actions  and  the  force  feedback.  With  increasing  distance  in  the  telesurgery  task,  time  delay 
will  become  a  problem  and  without  special  compensation,  will  produce  instability,  making 
telesurgery  impossible.  Communication  of  information  over  long  distance  involves  time. 
delays  associated  with  sequential  shift  register  operations  as  well  as  the  transmission  of 
radio  signals  through  space.  In  the  case  of  telesurgery,  (particularly  where  satellites  are 
used)  these  delays  appear  both  in  the  transmission  of  conunands  to  the  remote  surgical 
system  and  in  the  feedback  of  information  to  the  surgeon.  Among  all  the  effects  of  time 
delays,  such  as  image  transmission,  control  signal  transmission,  sensing  information 
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feedback,  and  so  on,  force  reflection  is  dominantly  important  to  perform  telesurgery 
practically.  The  latter  is  true  because  a  human  can  ignore  visual  time  delays  of  a  small 
fraction  of  a  second,  but  those  in  a  force  loop  inherently  close  the  loop  through  the  control 
handle  and  cannot  be  suppressed  by  the  human  operator  without  holding  the  control  handle 
rigid. 

The  force  reflecting  control  of  a  teleoperation  system  with  large  time  delay  (more  than 
0.2  second)  is  unstable  (Bejczy,1994).  Usually  the  coimnunication  time  delay  is  in  order 
of  1  second  if  satellite  communication  is  used  (Sheridan,1989).  In  that  case,  the  delayed 
forces  imposed  on  the  operator’s  (surgeon’s)  hand  will  be  significantly  out-of-phase  with 
his  or  her  intended  motions,  and  will  therefore  excite  instability  in  the  teleoperation  system 
and  make  teleoperation  or  telesurgery  very  difficult  to  implement.  From  a  control  system 
point  of  view,  large  time  delay  generates  a  large  phase  lag  in  the  system  loop,  decreasing 
the  phase  margin  of  the  system  significantly  and  making  the  system  unstable.  It  is  very 
difficult  to  design  compensator  for  such  a  delay  (Niemeyer  &  Slotine,1991).  Therefore,  the 
classic  controller,  such  as  proportional-plus-derivative  (PD)  control  can  not  make  the 
teleoperation  system  with  large  time  delay  stable  unless  some  way  can  be  found  to  change 
the  system’s  physical  parameters.  Fig.  2.9  shows  a  typical  dynamic  response  of  a 
teleoperation  system  under  force  reflecting  control  when  the  end  point  of  the  slave 
manipulator  contacts  the  surface  of  an  object.  The  instability  in  this  operation  is  shown 
clearly  in  the  dynamic  response. 


o  CO  <o  C3)  cvi  in  00  xj*  o 
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Fig.  2.9  Typical  Dynamic  Response  of  Force  Reflecting 

2.2.2.  Approaches  to  Cope  with  Time  Delay: 

A  conceptual  diagram  of  a  master-slave  system  for  telesurgery  is  shown  in  Fig.  2.10. 
Generally  speaking,  in  a  telesurgery  system,  there  are  a  master  manipulator  and  a  slave 
manipulator,  which  together  comprise  a  master-slave  manipulation  system.  To  implement 
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Fig  2.10  Conceptual  Diagram  of  Master-Slave  Systems 


telesurgery,  a  human  operator  (surgeon)  performs  his/her  task  for  the  patient  by  means  of 
the  master-slave  system  through  a  long  distance  communication  link.  This  may  result  in  a 
significant  communication  time  delay  between  the  master  manipulator  and  slave 
manipulator.  The  slave  manipulator  must  be  equipped  with  surgical  tools  to  perform  the 
operation,  and  the  master  manipulator  must  have  a  versatile  tool  handle  to  let  the  surgeon 
control  the  various  tools.  To  pursue  surgery  effectively,  some  computer-graphic  aiding  of 
vision  is  necessary,  including  superposition  of  target  and  predictor  icons  on  the  video 
image. 

Several  approaches  to  coping  with  time  delay  in  telerobotics  have  been  proposed  since 
the  1960s.  Ferrell  (1966),  Buzan  (1989)  and  Buzan  and  Sheridan  (1989)  investigated  the 
special  case  of  delayed  force  feedback,  and  developed  specialized  predictors  to  cope  with 
delayed  force.  Time  delay  in  a  teleoperation  loop  can  also  be  circumvented  altogether  by 
using  supervisory  control,  in  which  the  telerobot  is  programmed  to  perform  operations  in 
short  closed  loop  segments  and  control  is  managed  locally  through  artificial  sensors  and 
computer  within  the  task  segments  (Ferrell  and  Sheridan,  1967;  Brooks,  1979;  Conway  et 
al.,  1990;  Funda  et  al.,  1992;  Sheridan,  1992).  Hirzinger  et  al.  (1993)  demonstrated 
telerobotic  control  from  the  Earth  to  the  NASA  Space  Shuttle  for  simple  manipulation 
tasks,  making  use  of  both  predictor  display  and  supervisory  control  techniques.  Other 
investigators  have  suggested  means  of  artificially  damping  or  smoothing  control  signals  to 
prevent  instability  (Niemeyer  and  Slotine,  1991). 

So  far,  the  approaches  proposed  and  investigated  can  be  categorized  into:  (1)  “Move 
and  Wait”;  (2)  Predictor  display;  (3)  Supervisory  Control;  (4)  Passive  Compensation.  In 
our  first  year’s  research,  we  investigated  approaches  (1),  (2),  (3)  on  our  telemanipulation 
system  according  to  our  schedule.  Approach  (4),  passive  compensation,  will  be  studied  on 
our  system  in  the  fall,  1995.  Considering  that  a  human  operator  is  in  the  loop  in 
combination  with  signal  transmission  time  delay  and  environmental  uncertainty,  we 
proposed  a  new  approach:  “Fuzzy  Sliding  Control”.  The  following  sections  will  describe 
the  control  theory  and  the  experimental  results.  Another  approach,  supervisory  control, 
has  been  implemented  in  the  telemanipulation  system  and  will  also  be  described. 

To  seek  an  appropriate  approach  to  stable  controller  design,  we  need  to  simplify  the 
model  of  the  system  first.  Three  important  features  cmcial  to  controller  design  are:  timpi 
delay  between  the  master  and  slave  system;  uncertainty  of  the  task  environment;  and  the 
human  operator  in  the  force  control  loop.  Since  there  is  no  way  to  change  the  human  force 
reflection  and  the  time  delay  in  the  teleoperation  system,  we  can  only  model  the  task 
environment.  In  our  research,  a  simple  task  simulating  the  telemanipulator  interaction  with 
the  tissue  of  the  patient’s  body  was  provided  by  a  mechanical  beam.  Normal  forces  applied 
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are  non-linear  with  both  normal  displacement  and  tangential  position,  much  as  would  be 
encountered  in  surgery.  This  provided  a  good  test  of  our  methods  for  coping  with  delay. 
Fig.  2.1 1  shows  the  beam.  Interaction  with  the  beam  can  be  separated  into  three  different 
control  modes:  free  motion  mode,  transition  mode,  and  contact  mode.  Among  the  three 
control  modes,  the  operation  of  free  motion  mode  is  always  stable  because  the  loads  of 
motors  are  almost  the  same  in  this  case.  Therefore,  in  our  research,  we  focus  on  contact 
mode  and  transition  mode. 


2.2.3.  Theory  of  Fuzzy  Sliding  Control: 

From  our  own  trials  with  the  “move  and  wait”  approach  to  controlling  with  time  delay, 
we  found  that  the  delayed  forces  imposed  on  the  controlling  hand  perturb  the  system  with 
an  out-of-phase  signal  and  therefore  excite  instability  in  the  telemanipulation  system.  If  we 
can  apply  a  dead  zone  together  with  some  kind  of  logic  to  the  system  controller,  we  can 
decrease  the  effect  of  perturbation  of  the  delayed  force  feedback,  and  thus  can  improve  the 
stability  of  telemanipulation.  To  the  dead  zone  we  added  fuzzy  control.  Combining  with 
the  sliding  control  principle,  a  new  approach  we  call  Fuzzy  Sliding  Control  is  proposed. 
The  advantages  of  this  combination  are  given  below. 

Why  sliding  control  (with  deadzone)  and  fuzzy  control?  Fig.  2.12  illustrates  the  sliding 
control  principle,  basically  a  nonlinear  switching  control  strategy.  By  predefining  the 
switching  surface  and  a  dead  zone  or  boundary  layer  along  the  switching  surface,  the 
control  law  can  be  chosen  so  that  any  control  state  in  the  phase  plane  can  automatically 
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reach  the  boundary  layer.  Within  the  boundary  layer,  the  control  state  can  be  tuned  such 
that  it  reaches  the  switching  surface  without  overshoot.  Then  along  the  switching  surface 
the  control  state  converges  to  the  origin.  This  allows  for  acceptable  control  yet  prevents 
oscillation.  Sliding  control  is  absolutely  stable  according  to  the  Lyapunov  Stability 
Principle  (Slotine, 1991).  • 


2nd  Order  System:  x{t) = f(x,t)+u+d{t) 

Switching  Surface:  s(x,t)=e+X’e 

where  e=x(t)-x^{t) 

SMC  with  Boundary:  u=-Xe-K'satis/ 6) 

.  if  W<1 

Fig.  2.12  Principle  of  Sliding  Control 
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The  purpose  of  applying  fuzzy  control  in  teleoperation  is  to  achieve  an  adjustable  dead 
zone  control,  which  is  better  than  simple  dead  zone  control  because  of  its  smooth  nonlinear 
dead  zone  function.  To  achieve  fuzzy  adjustment  of  the  width  of  the  dead  zone  the  rules 
may  change  with  tasks,  and  for  simplicity  in  implimentation  three  rules  can  be  chosen: 

IF  the  perturbation  magnitude  is  big  (B)  THEN  the  width  of  dead  zone  is  big  (B); 

IF  the  perturbation  magnitude  is  small  (S)  THEN  the  width  of  dead  zone  is  small  (S); 

IF  the  perturbation  magnitude  is  zero  (Z)  THEN  the  width  of  dead  zone  is  zero  (Z). 

The  control  law  (which  determines  the  magnitude  of  the  control  signal)  may  also  be 
fuzzified  (Fig.  2.13).  In  a  one  dimensional  case  (only  error  input),  for  example,  we  could 
have  fuzzy  subsets  Positive  Small  (PS),  Zero  (Z),  and  Negative  Small  (NS).  Then  we 
might  have  the  following  control  rules: 

IF  the  error  input  is  Z,  THEN  the  control  output  is  Z; 

IF  the  error  input  is  PS,  THEN  the  control  output  is  PS; 

IF  the  error  input  is  NS;  THEN  the  control  output  is  NS; 


Fig.  2.13  Fuzzy  Control  with  Dead  Zone  (in  1  dim.) 

Using  the  above  rules  and  procedures,  the  controller  can  provide  a  smooth  output 
function.  For  realistic  application  in  teleoperation,  the  fuzzy  inference  is  two  dimensional 
(i.e.  two  inputs:  error  and  derivative  of  error)  and  the  control  rules  must  be  designed  in  the 
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2-dimensional  phase  plane.  Once  the  control  rules  have  been  chosen,  it  is  necessary  to  tune 
the  control  rules  in  the  phase  plane  to  make  them  consistent.  To  do  this,  the  Fuzzy  Sliding 
Control  Law  has  been  derived  based  on  the  general  sliding  control  principle.  Figures  2.14 
through  2.17  describe  the  fuzzy  sliding  control  principle,  the  design  methodology  and  its 
realization  in  the  frame  of  fuzzy  control  system  structure. 

This  rather  ad  hoc  procedure  for  the  controller  includes  the  following  steps: 

1 .  Set  up  the  preliminary  control  rules  from  intuition  about  the  task  dimensions; 

2.  Choose  the  switching  line  and  boundary  layer  thickness; 

3 .  Choose  the  fuzzy  partitions  (membership  functions)  of  the  control  region  and 
boundary  layer,  and  those  for  the  input  variables  and  control  variable; 

4.  Modify  the  control  rules  according  to  the  Fuzzy  Sliding  Control  Law; 

5 .  Implement  the  fuzzy  sliding  control  on  the  general  fuzzy  control  system  structure; 

6 .  Simulate  and  evaluate  the  control  system; 

7.  Re-iterate  from  step  2  until  satisfactory  responses  are  obtained. 


Phase  Plane  of  Fuzzy  Sliding  Control  with  Fuzzy  Soft  Boudary 


Control  Law  for  FSMC: 


u=-X'e- K j^^e,eyX,5)‘  sat{s !  5) 


oc  \s\=e-\rX‘e 


Fig.  2.14  Fuzzy  Sliding  Control 
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2.2.4.  Supervisory  Control  of  Telesurgery  System: 

Supervisory  control  is  another  approach  for  dealing  with  time  delay  in  a  teleoperation 
system,  which  was  initially  developed  in  the  Human-Machine  Systems  Laboratory  at  MIT 
in  the  1960s  and  1970s.  Ferrell  and  Sheridan  (1967)  proposed  that  a  hierarchical  system 
based  on  a  human  supervisor— computer  subordinate  relationship  could  be  used  in 
teleoperation  to  solve  some  of  the  control  problems  involving  time  delay.  Under 
supervisory  control  (Fig.  2.18)  the  operator  sends  coded  instructions  to  the  remote 
telerobot  specifying  subgoal  conditions  and  procedures  with  reference  to  sensor  states  that 
must  be  satisfied.  The  operator’s  input  could  range  from  a  purely  manual  analog 
command,  demonstrating  in  time  and  space  what  is  intended,  to  a  highly  abstract  symbolic 
command  made  up  of  alphanumeric  keystrokes.  These  instructions  are  sent  open  loop 
relative  to  the  human.  The  remote  computer  then  interprets  the  message  and  acts  on  the 
sensor  information  available  to  it  about  its  own  environment.  In  this  way  the  control  loop 
is  closed  within  the  remote  site,  where  there  is  no  time  delay.  The  remote  computer  then 
relays  that  information  which  is  deemed  important  and  necessary  for  effective  supervision 
back  to  the  operator,  the  responsibility  for  the  specific  details  of  control  being  left  to  the 
subordinate  computer. 

The  shortcomings  of  supervisory  control  with  time  delay  are  the  shortcomings  of  the 
human-machine  communication  interface  on  the  supervisor  side.  Fig.  2.19  shows  our 
supervisory  control  system.  There  are  two  manipulators  (master,  slave),  two  PCs  (one  for 
master  control,  one  for  slave  control),  and  an  SGI  workstation  at  the  supervisor’s  interface. 
Force  feedback  is  also  reconstructed  to  aid  the  supervisor’s  operation. 

As  noted  above,  human-computer  communication  in  supervisory  control  can  be  in 
terms  of  analog  (space-time  movement)  language  or  in  symbolic  (alphanumeric)  language, 
or  some  combination.  Symbolic  language  is  not  realistic  for  most  commands  in  telesurgery 
at  the  current  level  of  technology,  so  we  rely  mostly  on  master-slave  analog  demonstration 
as  a  means  for  the  surgeon  to  communicate  his  intentions  to  the  computer  and  the  human 
assistant.  The  analog  commands  (intended  kinematic  positions  and  movements)  are 
displayed  as  computer  graphics  by  the  SGI  (a  virtual  reality  superposed  on  the  video  image 
of  the  actual  patient  scene)  which  are  observable  at  both  master  and  slave  ends.  The  fact 
that  our  master  and  slave  manipulators  have  identical  kinematics  is  helpful  in  this  regard.  A 
computer  keyboard  is  used  to  input  and  edit  some  commands.  The  simulated  slave 
environment  can  also  supply  force  feedback  to  the  surgeon’s  (supervisor’s)  hand.  In  this 
way,  the  surgeon  can  both  oversee  the  surgery  operation  process  and  feel  the  physical 
interactions.  Because  of  the  time  delay,  the  supervisor  can  not  intervene  in  the  operation  of 
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the  remote  slave  manipulator  instantly,  and  therefore  it  is  important  that  the  slave  computer 
have  an  appropriate  interface  to  the  assistant  who  is  nearby  the  patient.  In  this  way  the 
assistant  can  monitor  the  force/position  control  and  intervene  whenever  necessary. 


Fig.  2.18  System  Structure  of  Supervisory  Control 


Master  Robot  1  Slave  Robot 
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Fig.  2.19  Supervisory  Control  System  of  Telesurgery 


2.2.5.  Preliminary  Human-in-the-loop  Experiments  with  Telesurgery  under 
Time  Delay  using  a  Virtual  Remote  Environment: 

For  the  purpose  of  preliminary  experiments,  in  which  we  had  better  control  over  key 
parameters  of  the  telerobot  system,  time  delay  and  manipulation  tasks,  we  developed  a 
virtual  slave  arm  (which  includes  dynamic  simulation  of  the  motor  driving  system),  a 
virtual  beam  to  probe,  and  the  associated  graphics  displays.  The  task  in  each  case  was  to 
move  the  real  master  arm  so  that  the  virtual  slave  arm  contacted  the  virtual  beam  at  a  given 
point  from  a  free-space  starting  position,  apply  a  small  force,  then  slide  along  the  beam  to 
another  specified  point  while  maintaining  force  contact. 

Using  the  master-virtual  slave  system  (Fig.  2.20),  the  following  control  techniques 
were  studied; 

•  Fuzzy  Sliding  Control; 

•  Sliding  control  with  boundary  layer; 

•  Adaptive  dead  zone  control; 

•  Traditional  PD  control; 

Time  delays  of  0,  100,  200,  500,  800  and  1000  ms  were  tested.  Dynamic  data  were 
recorded  and  used  to  do  objective  evaluation  based  on  the  vibration  magnitude.  Subjective 
difficulty  evaluations  were  also  made  (Fig.  2.21).  Vibration  magnitude  results  are  shown 
in  Fig.  2.23.  Comparisons  of  evaluations  (both  subjective  and  objective)  of  the  controllers 
can  be  found  in  Fig.  2.22  and  Fig.  2.24. 
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Fig.  2.21  Subjective  Evaluation  of  Controllers  with  Time  Delays 


Average  Amplitude  of  Vibration 


1.5 


Fig.  2.24  Comparison  of  Objective  Evaluations 
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2.2.6.  Preliminary  Human-in-the-Ioop  Experiments  with  Telesurgery  under 
Time  Delay  using  an  Actual  Remote  Environment: 

After  the  full  bilateral  master-slave  system  was  set  up  (Fig.  2.1),  experiments  were 
carried  out  by  means  of  a  real  beam  instrumented  with  strain  gauges.  Fig.  25  shows  some 
dynamic  responses  (force  feedback  signals)  of  a  typical  subject  experiment.  The 
experimental  results  are  consistent  with  the  data  obtained  on  the  mater-virtual  slave  system. 
They  show  significant  improvement  of  operator  performance  in  the  contact  mode  (Fuzzy 
Sliding  Control  over  PD  Control). 

(a)  FSC  (1  Sec  Delay) 


Time  (sec) 


(b)  PD  Control  (1  sec  Delay) 
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Fig.  2.25  Force  Feedback  of  Teleoperation  with  1  sec 
Time  Delay  (FSC  &  PD  Control) 
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2.3.  EXPERT-NOVICE  COOPERATIVE  TELESURGERY 


2.3.1  Development  of  experimental  tasks: 

The  challenge  here  is  to  develop  credible  means  to  evaluate  various  techniques  of 
cooperative  manipulation  (as  previously  defined)  in  the  context  of  telesurgery.  In  such 
research  one  faces  the  dilemma  of  choosing  between  experiments  too  abstract  to  be  credible 
to  the  practitioner  versus  those  too  specific  to  a  given  task  to  be  scientifically 
generalizable.  Too  specific  a  task  might  be  using  a  probe  simply  to  judge  accuracy  in 
following  geometric  figures.  Further,  at  this  early  stage,  animal  and  clinical  trials  were  not 
options.  In  this  project,  focusing  on  laparoscopy  provides  a  manageable  scope  within 
which  to  work  and  as  a  compromise,  the  experimental  tasks  that  will  be  evaluated  are  basic 
training  exercises  used  by  surgeons  learning  laparoscopic  techniques. 

A  set  of  these  tasks  is  described  by  Steele,  Hosking  and  Chung  (1994).  The  tasks 
provide  experience  in  manipulating  all  of  the  usual  tools  used  in  laparoscopy  in  the 
somewhat  realistic  environment  of  a  laparoscopy  trainer.  The  tasks  include:  grasping  with 
forceps,  precision  motion  of  the  tools,  use  of  the  scissors,  use  of  laparoscopic  clip 
appliers,  various  suturing  techniques  and  a  comprehensive  task  on  animal  tissue.  Some  of 
the  tasks  have  adapted  for  our  use  and  are  described  in  table  2.1.  Suturing  has  been 
omitted  because  as  a  two  handed  task,  a  surgeon  cannot  perform  it  with  our  system. 
However,  with  the  advent  of  clip  appliers  and  staplers,  most  of  the  uses  of  sutures  can  be 
duplicated.  In  addition,  there  is  work  underway  to  develop  automatic  suturing  tools  which 
require  the  use  of  only  one  hand  (Faraz,  Payandeh,  Nagy,  1995). 

To  study  different  forms  of  interaction,  in  a  given  experimental  run,  the  test  tele¬ 
surgeon  will  be  assigned  one  of  the  laparoscopic  tools  used  in  each  task.  The  assistant  will 
manipulate  the  others.  For  example,  the  surgeon  might  control  the  laparoscope,  and  direct 
the  assistant  to  manipulate  the  other  tools  properly.  The  procedure  for  a  full  experimental 
run  will  be  described  later. 
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Table  2.1:  Elemental  training  tasks 


Task 


Grasp  & 
T  ransfer 


G  &  Twith 
Orientation* 


Precision 

Motion 


Use  of 
Scissors* 


Use  of  Ciip 
Appiier 


Tools  required 

Description 

Errors 

2  grasping  forceps, 

laparoscope 

Grasp  dental  swap  from 

cup  1  with  forceps. 

Transfer  to  2nd  pair 

forceps.  Place  in  cup  2. 

Dropping  swab 

Grasping  wrong 

swab 

2  grasping  forceps, 

laparoscope 

As  G&T,  but  replace 

swabs  with  paperclips, 

clips  must  be  grasped 

along  long  side,  not  at 

ends. 

Dropping  clip 

Grasping  wrong 

clip 

Grasping  ciip  at 

end 

2  grasping 

forceps, 

laparoscope 

Replace  clips  with  small 

folded  pieces  of  paper. 

Grasp  paper  from  cup  1 . 

Tear  along  fold.  Place 

pieces  in  cup  2. 

Dropping  paper 

Grasping  wrong 

piece 

Incorrect  tear 

scissors, 

grasping  forceps, 

laparoscope 

Grasp  end  of  suture  material/ 

string  with  forceps.  Cut 

beneath  grasp  point.  Place 

cut  piece  in  cup. 

Cutting  wrong  string 

Dropping  cut  piece 

clip  appiier, 

scissors, 

laparoscope 

Apply  1  clip  on  either  side  of 

mark  at  middle  of  rubber  band 

wrapped  around  cup.  Cut 

band  between  clips 

Incorrect  placement 

of  clips 

Cutting  outside  of 

clipped  region  of  band 

2  dissecting  forceps, 

clip  appiier, 

scissors, 

laparoscope 

Separate  tissue  near  vein  from 

vein  with  forceps.  Apply  clips 

to  section  of  vein.  Cut 

between  clips 

Damaging  vein 

Misapplying  clips 

Incorrect  cut 

*  see  Fig.  2.7  for  apparatus 
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2.3.2  Preliminary  cooperation  experiment:  format,  results  and  discussion: 

A  simple  test  to  evaluate  the  utility  of  cooperation  between  a  remote  expert  and  a  local 
assistant  was  performed  in  the  fall  of  1994.  (Ottensmeyer,  1995).  In  this  test,  the 
participants  were  instructed  to  assemble  geometric  puzzle  pieces  (Tangrams)  on  a  table, 
according  to  a  given  pattern.  The  expert  was  given  the  pattern,  and  instructed  the  assistant 
in  how  to  assemble  it.  The  expert's  mode  of  instruction  was  either  verbal  (symbolic)  only, 
or  symbolic  with  the  addition  of  an  analog  device— a  pointer,  representing  a  simple 
teleoperator.  Using  this  equipment,  the  experiment  can  be  compared  with  tele-mentoring, 
as  described  by  Kavoussi,  et  al.  (1994). 

What  was  clearly  shown  was  that  the  addition  of  the  pointer  device  in  the 
communication  both  reduced  the  time  required  to  complete  the  pattern  by  about  44%  (table 
2.2)  and  reduced  the  number  of  errors  incurred  in  positioning  the  individual  puzzle  pieces 
in  X  and  y  directions  by  about  80%.  There  was  no  improvement  in  the  orientation  error  rate 
(pieces  in  correct  location  but  incorrect  in  rotation),  but  this  is  thought  to  be  due  primarily 
to  the  lack  of  experience  of  the  "expert".  It  was  noticed  that  with  proper  use  of  the  pointer, 
orientation  errors  were  virtually  eliminated,  but  few  of  the  "experts"  learned  the  strategy  in 
the  course  of  the  test. 

The  conclusions  that  one  can  draw  from  this  simple  test  are  that  closer  cooperation  (i.e. 
more  direct  involvement  of  the  expert)  improved  performance,  and  that  with  proper 
instruction,  errors  can  be  minimized,  implying  that  the  expert/surgeon  must  have  extensive 
experience  directing  assistants. 


Table  2:  Preliminary  cooperation  experiment  results 


Verbal  communication  only 

Verbal  with  pointer 

Completion  Time 

92.38  ±30.19  sec. 

52.13  ±26.12  sec. 

Position  Error  Rate 

1.46  ±  1.16  errors/min. 

0.28  ±  .79  errors/min. 

Orientation  Error  Rate 

2.71  ±  1 .71  errors/min. 

2.90  ±  2.43  errors/min. 

2.3.3  Sample  surgical  task  pre-test  results: 

Even  before  the  actuated  laparoscopy  tool  was  complete,  it  was  possible  to  use  the 
master-slave  system  to  control  the  CCD  laparoscope,  so  one  partial  set  of  tasks  was  run. 
This  mode  is  similar  to  the  clinical  tests  being  done  by  Kavoussi,  et  al.  (1994),  which  they 
call  tele-mentoring.  The  surgeon  controls  the  laparoscope  remotely  and  gives  instructions, 
but  the  assistant  performs  all  of  the  manipulation  tasks. 
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Time  (seconds) 


Fig.  2.26  shows  the  times  required  to  complete  each  individual  task  in  a  series.  Three 
tasks  (grasp  and  transfer,  G  &  T  with  precision,  and  use  of  scissors)  were  tried.  In  this 
early  case  both  the  “surgeon”  and  assistant  were  fairly  in  experienced.  As  such,  no 
conclusions  should  be  drawn  from  the  data,  but  a  cursory  examination  seems  to  indicate 
that  the  learning  curve  seems  to  fairly  steep.  Further  discussion  of  the  experiments  and  the 
plans  for  the  formal  tests  is  taken  up  in  the  next  chapter. 


Fig.  2.26:  Task  Completion  Times 


0  2  4  6 


Trial  Number 
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3.  PLANS  FOR  SECOND  YEAR 


3.1.  HUMAN-IN-THE-LOOP  TELESURGERY  SIMULATION 

Testing  and  demonstrating  our  solutions  to  the  time  delay  and  cooperative  manipulation 
problems  will  require  formally  defining  the  Communications/Time  Delay  block  shown  in 
the  system  diagram,  Fig.  2.1. 

As  mentioned  in  various  sections  above,  achieving  a  variable  time  delay  will  be 
accomplished  in  two  ways.  The  command  signals  between  the  master  and  slave 
manipulators  will  be  buffered  in  either  the  master  or  slave  Pentium,  and  transmitted  after  a 
given  time  has  elapsed.  Delay  in  audio  and  video  communication  will  be  accomplished 
through  the  use  of  commercial  equipment  designed  specifically  for  this  purpose. 
Equipment  like  this  is  typically  used  by  television  and  radio  broadcasters  to  prevent  the 
transmission  of  inappropriate  language  and  to  ensure  synchronization  of  signals  transmitted 
by  satellite. 

To  permit  the  transportation  of  either  the  master  or  slave  manipulator  to  a  site  remote  to 
the  laboratory,  we  will  either  be  adapting  commercially  available  video-conferencing 
equipment  for  use  with  our  system  or  making  use  of  the  internet.  The  video-conferencing 
mode  combines  three  ISDN  telephone  lines  to  yield  a  384  kbit/sec  channel,  sufficient  to 
transmit  good  quality  video  and  audio  signals,  and  to  carry  the  data  requirements  of  the 
master-slave  system.  In  theory,  the  bit  rate  through  the  internet  is  larger,  but  due  to 
latencies  and  delays  inherent  in  it,  will  likely  make  force  feedback  impractical. 

3.2.  COPING  WITH  TIME  DELAY 

To  complete  our  study  of  the  practical  approaches  for  force  reflecting  control  of 
teleoperation  system  thoroughly,  we  plan  to  investigate  one  additional  approach:  Passive 
Compensation.  We  will  conduct  further  human-in-the-loop  experiments  to  test  all  the 
approaches  under  different  time  delays,  collecting  both  dynamic  response  data  and 
subjective  evaluations.  We  will  compare  the  approaches  to  find  out  which  is  the  best 
approach  under  what  circumstance. 

3.2.1.  Passive  Compensation  Algorithms: 

There  are  two  similar  methods  for  passive  compensation,  which  are  based  on  passive 
communication  laws.  The  first  method  was  developed  by  Anderson  and  Spong  (1988) 
using  scattering  theory  (Johnson,  1950).  In  1990,  Niemeyer  and  Slotine  developed  the 
second  approach,  the  Wave  Variable  or  Energy  approach.  Both  methods  stem  from  the 
central  idea  of  the  Scattering  Theorem  which  derives  from  the  fact  that  an  analog  electrical 
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transmission  line  delays  the  signal  and  is  inherently  passive,  and  any  passive  two-port 
electrical  transmission  line  is  stable.  The  Scattering  Theorem  states  that  a  two  port  network 
is  passive  if  and  only  if  the  norm  of  its  scattering  operator  S  can  be  defined  for  a  two  port 
network  by  the  relationship  between  force  f  and  velocity  v, 

F-V  =  S(f  +  v) 

where  S  is  a  matrix  in  the  frequency  domain. 

Fig.  3.1  is  the  block  diagram  of  a  passive  two-port  network  used  as  a  communication 
link.  Controller  design  based  on  passive  compensation  requires  one  to  obtain  controller 
parameters  which  make  the  network  be  passive  (i.e.  satisfy  the  scattering  theorem).  The 
passive  compensation  method  guarantees  asymptotically  stable  operation  in  spite  of  any 
time  delays,  assuming  the  human  operator,  master,  slave  and  environment  can  all  be 
represented  as  passive  systems.  Anderson  and  Spong  (1988),  and  later  Niemeyer  and 
Slotine  (1990),  showed  that  the  stiffness  between  master  and  slave  was  effectively  reduced 
by  the  passive  communication  law.  This  reduction  of  stiffness  and  the  position  drifting 
problem  of  velocity-force  control  (Van  der  ham,  1995)  will  be  investigated  in  October  and 
November  of  1995. 


Dissipation  Time  Delay  Dissipation 


Fig.  3.1  Passive  Compensation  of  a  Communication 
Process  in  Teleoperation 

A  position-derivative  (PD)  controller  will  be  designed  for  the  motor  control  of  the  slave 
manipulator.  To  keep  the  dynamic  and  steady  states  identical  in  slave  and  master 
manipulator,  the  controller  type  of  slave  and  master  should  be  the  same.  Selecting  the 
gains  for  master  and  slave  controllers  can  be  done  partly  by  analysis  and  partly  by 
experiment.  For  force  fidelity,  the  position  controller  gains,  e.g.  Ks  and  Bs  should  be 
selected  to  make  the  system  as  stiff  as  possible.  This  comes  at  the  expense  of  a  larger 
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apparent  inertia  and  increased  damping  in  the  system.  Thus  there  is  a  trade-off  when 
adjusting  the  gains  for  the  passive  communication  law.  Lawn  (1992)  gives  gain  and 
parameter  settings  appropriate  for  such  experiments. 

3.2.2.  Demonstration  of  Various  Control  Approaches: 

Using  different  control  techniques,  namely  Fuzzy  Sliding  Control,  Predictor  Display, 
Move-and-Wait,  Supervisory  Control,  Passive  Compensation  etc.,  we  will  implement 
further  human-in-the-loop  teleoperation  experiments  on  our  established  PHANToM  master- 
slave  system  with  different  time  delays  and  a  real  beam. 

We  will  also  perform  laparoscopic  telesurgery  tasks  and  measure  performance  under 
different  time  delays.  These  experiments  will  be  discussed  in  the  next  section. 

3.3.  EXPERT-NOVICE  COOPERATIVE  TELESURGERY 

At  the  current  time  we  expect  to  run  two  complete  sets  of  experiments:  (i)  telesurgery 
without  time  delays  and  (ii)  telesurgery  with  time  delays.  These  experiments  will  be  done 
over  short  distances  within  the  lab,  but  will  be  amenable  for  longer  distance  demonstrations 
through  the  use  of  the  telecommunications  link.  It  is  hoped  that  it  will  be  possible  to  move 
the  master-side  system  to  a  local  hospital  for  demonstration  purposes. 

The  first  set  of  experiments  will  permit  us  to  develop  guidelines  for  use  in  cases  when 
delays  eire  negligible,  while  the  second  will  indicate  what  changes  would  be  necessary 
given  delays  in  communication. 

3.3.1  No  delay  experiments: 

September  and  October  1995  will  be  spent  finalizing  the  form  of  the  experiments— 
pretesting  the  tasks  and  developing  the  dexterity  tests.  Any  problems  with  the  equipment 
will  be  solved  during  this  period.  Tests  with  naive  subjects  will  be  conducted  in 
November.  As  an  "expert",  either  Dr.  James  Thompson,  Mark  Ottensmeyer  or  both  will 
become  familiar  and  well  practiced  at  the  various  tasks  during  the  pre-test  period.  It  is 
assumed  that  a  real  surgeon  working  through  a  telesurgery  system  would  have  extensive 
experience  with  it,  and  be  far  along  the  learning  curve  with  respect  to  its  operation. 

Subjects  will  first  perform  the  dexterity  tests  and  go  through  a  pretest  interview.  These 
results  will  yield  data  that  may  permit  us  to  determine  if  performance  can  be  predicted  based 
on  objective  measures  other  than  direct  tests  with  laparoscopic  tasks.  Then  one  of  the 
laparoscopic  instruments  will  be  assigned  to  the  surgeon,  the  assistant  being  given  the  other 
two,  and  the  series  of  tests  will  be  performed.  A  series  will  include  a  set  of  each  of  the 
different  elemental  tasks  with  order  counterbalanced  to  prevent  learning  bias.  If  practical 
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we  will  assign  the  tools  and  have  the  subject  perform  each  task  again  to  allow  inter-subject 
comparison. 

In  the  no-delay  case  the  surgeon  and  assistant  will  be  separated  by  a  distance  of  only  a 
few  feet  and  by  a  curtain  so  that  the  surgeon  will  not  have  direct  vision  of  the  "remote"  site. 
The  same  video  image  will  be  presented  to  each,  and  direct  audio  communication  will  be 
available  to  the  pair.  This  arrangement  provides  all  the  separation  of  surgeon  and  assistant 
necessary  to  evaluate  tele-cooperation,  despite  the  short  distance. 

The  tests  will  be  recorded  by  two  video  tapes:  one  from  the  CCD  laparoscope  and  one 
from  an  external  camcorder.-  Timing  and  errors  will  be  determined  from  the  tapes,  and 
analysis  of  these  data  will  occur  in  November  and  December  and  will  contribute  to  the  any 
modifications  necessary  for  the  time  delay  experiments  in  early  1996. 

3.3.2  Time  delay/remote  venue  experiments: 

In  the  first  few  months  of  1996  we  will  prepare  and  conduct  experiments  to  evaluate  the 
effects  of  time  delay  on  the  surgeon-assistant  interaction.  Given  these  results,  different 
recommendations  may  be  made  on  how  best  to  coordinate  tasks  for  different  levels  of 
delay. 

The  experiments  themselves  will  be  very  similar  to  those  used  in  the  no-delay  case. 
Assistants  will  go  through  the  dexterity  tests  and  interview,  and  tools  and  tasks  will  be 
assigned  in  the  same  way.  Keeping  experimental  format  the  same  except  for  the  delay 
aspect  will  permit  us  more  easily  to  compare  the  two  cases. 

Based  on  the  findings  of  the  simplified  environment  time  delay  experiments,  one  or  two 
of  the  time  delay  solution  schemes  will  be  used  in  the  laparoscopy  task  experiments. 

3.3.3  Paramedic  Skill  Evaluation/Modelling  in  Cooperative  Telesurgery: 

There  are  four  major  variables  here:  the  surgeon,  the  assistant,  the  diagnostic/ 

therapeutic  task,  and  the  time  delay.  In  practice,  the  assistant  would  likely  be  a  paramedic 
who  will  have  some  experience  with  trauma  patients,  but  will  not  have  the  diagnostic 
ability,  knowledge  base,  surgical  skill  and  medical  decision  making  ability  of  the  surgeon. 
Different  paramedics  will  have  different  strengths  and  weakness. 

It  has  been  suggested  that  if  these  strengths  and  weakness  can  be  quantified  (e.g.  by 
visuo-spatial,  manual  dexterity,  personality,  and  other  tests),  then  combined  with  the 
circumstances  of  a  given  procedure,  a  surgeon  will  be  able  to  look  up  in  a  database  the  best 
way  to  allocate  tasks  between  him  or  herself  and  the  paramedic.  To  develop  such  a 
database  would  require  a  model  of  the  interaction  which  would  use  test  results  and 
circumstance  as  inputs  and  a  near  optimal  task  assignment  scheme  as  an  output.  Using  the 


page  40 


results  of  our  experiments  it  may  be  possible  to  begin  to  develop  such  a  model,  given  the 
test  results  of  the  subjects,  and  their  subsequent  performance  in  the  surgical  training  tasks. 


3.3.4  Data  analysis  and  reporting: 

Analysis  of  the  fall  '95  experiments  will  begin  in  November  '95  and  continue  until 
complete  (expected  January  '96  at  the  latest).  The  winter  '96  tests  will  likely  be  evaluated 
beginning  in  March  or  April  '96.  Based  on  this  information,  conclusions  and  guidelines 
will  be  drawn  up  covering  the  interaction  given  various  time  delays  and  subject 
competency.  Final  reports  will  be  complete  in  the  summer  of  1996,  and  any  papers  will  be 
under  way  by  then. 

Simple  demonstrations  of  the  equipment  are  ready  now,  and  will  become  more 
extensive  as  the  experiments  go  ahead.  One  of  the  final  demonstrations  will  include  an 
animal  tissue  dissection  test,  likely  under  various  combinations  of  surgeon-assistant 
cooperation. 


4.  CONCLUSIONS 

After  the  first  year’s  effort,  significant  steps  have  been  made  in  pursuit  of  a  solution 
to  the  time  delay  problem  and  in  developing  guidelines  for  the  interaction  between  tele¬ 
surgeon  and  local  assistant.  A  human-in-the-loop  tele-surgery  simulation  system  has  been 
developed,  including  a  force  reflecting  master-slave  system,  a  novel  remotely  controlled 
surgical  tool  and  test  apparatus  to  provide  objective  measures  of  performance.  Various 
control  schemes  to  provide  stable  force  feedback  during  teleoperation  have  been  studied, 
including  Fuzzy  Sliding  Control,  and  will  be  implemented  in  the  coming  months  for  use  in 
experiments  based  on  surgical  training  tasks.  The  next  year’s  work  will  include 
experiments  with  novice  assistants  to  determine  how  best  to  allocate  tasks  between  tele¬ 
surgeon  and  assistant,  in  both  no-delay  cases,  and  cases  with  delay,  using  the  best  of  the 
control  schemes.  To  demonstrate  proof  of  concept,  a  tele-communications  method  will  be 
used  to  permit  the  locales  of  the  surgeon  and  the  task  to  be  separated  by  non-trivial 
distances. 
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APPENDIX  A:  FUZZY  LOGIC  CONTROL 


Fuzzy  logic  is  much  closer  in  spirit  to  human  thinking  and  natural  language  than 
traditional  logic.  Essentially  it  provides  an  effective  means  of  capturing  the  approximate, 
inexact  nature  of  the  real  world.  A  fuzzy  logic  controller  (FLC)  is  a  set  of  linguistic  control 
rules  which,  in  combination,  result  in  an  algorithm  to  map  quantitative  inputs  to  quantitative 
outputs.  The  methodology  of  the  FLC  is  very  useful  when  the  processes  are  too  complex 
for  analysis  by  conventional  quantitative  techniques  or  when  the  available  source  of 
information  can  only  be  interpreted  qualitatively,  inexactly,  or  uncertainly.  Thus  fuzzy  logic 
control  may  be  viewed  as  a  step  toward  a  rapprochement  between  conventional  precise 
mathematical  control  and  human-like  decision  making. 

Here  is  an  example  of  FLC  with  approximate  inference  (Mamdani’s  Approach).  If  there 
exist  N  rules  in  the  fuzzy  control  system,  the  ith  rule  is  in  the  form  of 
IF  inputs  Xi  =  Aj ,  X2  =  Bj ,  THEN  u  =  Q  . 

Let  Xio,  X20  be  the  inputs  of  Xj,  X2  correspondingly,  so  the  truth  value  of  the  premise  (or 
membership)  can  be  expressed  as 
Ih  =  M-AiC^lO  )  A  llBi(X2o) 

where  |XAi(xio  )  is  the  membership  of  Xio  pertaining  to  fuzzy  set  Ai  and  |J.Bi(x2o)> 
membership  of  X20  pertaining  to  Bj,  and  “a”  is  an  operator,  which  can  be  defined  as 
“minimum”.  Then,  the  membership  of  the  control  output  is 

where  “v”  is  an  operator,  which  can  be  defined  as  “maximum”.  Therefore  the  output 
(fuzzyfication,  or  fuzzy  decision  making)  can  be  obtained  from  the  centroid  formula. 

The  inference  process  can  be  described  in  the  following  figure. 
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Fig.  AP-1  Inference  Process  of  Fuzzy  Control 


z 


page  46 


